Apparatus and Methods of Computer-Simulated Three-Dimensional Interactive Environments

ABSTRACT

Computer-simulated three-dimensional environments include automatable and/or constrainable camera control, for video gameplay, real estate and/or landscape demonstrations, or any other digitizable environment. In gameplay applications, the system can be customizably programmed to automatically adapt to the environment based on the player&#39;s location within the virtual environment, information about what the programmer believes is relevant (or wants to make relevant) in the scene being displayed, and other factors. Certain embodiments of the inventive apparatus and methods generally automatically incorporate and honor the “rules of cinematography,” but also preferably include other “action video game” principles that can override or trump those rules.

FIELD OF THE INVENTION

The present invention relates generally to computer-simulatedthree-dimensional environments, such as can be generated and displayedinteractively on computer monitors or similar displays. Morespecifically, the invention relates to systems having automatable and/orconstrainable camera control within an interactivecomputerized/digitized environment. Among other applications, theinvention is useful for video gameplay, real estate and/or landscapedemonstrations, or any other digitizable environment. As a new gameplayfeature for video games (such as third person games and the like), thecamera system invention can be customizably programmed to automaticallyadapt to the gameplay environment based on the player's location withinthe virtual environment, information about what the programmer believesis relevant (or wants to make relevant) in the scene being displayed,and other factors. Among other things, the invention can enhance suchgameplay by allowing the user to focus on playing the game, rather thanhaving to also worry about the complexity of controlling the camera andthe corresponding view being displayed to the user.

Certain embodiments of the inventive apparatus and methods generallyautomatically incorporate and honor the “rules of cinematography,” butalso preferably include other “action video game” principles that canoverride or trump those rules. For example, in certain embodiments ofthe technology within an action video game, programmers will not want totake liberties that are taken when the rules of cinematography are usedin movies (such as removing certain objects from a camera shot orautomatically adjusting the position of one of the people within thecamera shot). If applied to certain high/fast action video games, those“movie” liberties (dictated by strict adherence to the “rules ofcinematography”) would disrupt the gameplayer's immersion into thevirtual world, rather than more closely mimicking actual physicalrealities.

BACKGROUND OF THE INVENTION

As with most or all computer technologies, the use and complexity ofgraphical, digital “environments”, and the ability to allow users tointeract within those simulated environments, have evolved significantlyover the years. Applications of such technologies and systems are quitevaried (including, by way of example and not by way of limitation, 3DCAD programs, architectural modeling software to provide virtual toursof actual or virtual homes or landscapes, and others). Among otherthings, this evolution has included the ability to create and “travel”through much more complicated and much more realistic virtual worlds, atmuch faster “speeds” than were achievable even a few years ago.

Among the many examples of those technologies are computer video games.Early two-dimensional games such as Pong required only basic input froma user/player, such as moving to the right or the left on the screen. Aspart of this evolution, other controls were added (such as guns or otherweapons, thrusters or other ways to move things on the screen acrossboth of the two dimensions of the video display, etc.).

These games further evolved to include three-dimensional (3D)experiences. Such video game methodology typically includes using a“camera”, or a displayed point of view in the digital three-dimensionalworld of the game. This displayed point of view or camera typically iscontrolled by the user, to select and manipulate the view displayed onthe video monitor or other display during gameplay.

Three-dimensional games typically are either played in the first personor third person. In a first person game, the camera takes the positionof the “eyes” of the player. In a third person game, the camera displaysboth the player's character and the surrounding environment, and theuser (the human being playing the game) views the action on the screenfrom the perspective of a “third person”, rather than viewing itdirectly through the eyes of one of the characters in the game.

In third person video games, the camera system apparatus and methodsconventionally have used either (a) fixed camera positions that changeas the player moves from one scene or location to another within thedigital environment, or (b) a user-controlled camera which is positionedslightly above and behind the player. Between those two, the latterapproach typically can provide a more dynamic and engaging userexperience (such as by simulating the need for, and/or the effect of theplayer to turn his head to the right or left or to look up or down, orotherwise feel more immersed within the digital environment). However,that latter approach typically requires that the player has to not onlymove his character throughout the digital environment (and shoot weaponsor take other actions such as jumping, swinging his arms or kicking hislegs, etc.), but also manipulate controls to adjust the camera positionor viewpoint, via an additional controller or input mechanism.

In many such video games, the player typically is in control of thegameplay, to at least some degree. As part of that control, the player'scontrol of the camera typically not only affects the aesthetics of thegame experience, but also can be a means by which the player interactsin the virtual world. Because the main focus of the player typically isachieving various game objectives (i.e. navigating around objects,attacking enemies, jumping from platform to platform) in order toadvance and/or score well within the game, some game designs have tendedto emphasize the most functional view of the gameplay action, withlittle regard to the aesthetics.

Although computer memory, graphics capabilities, processing speed, andother “limits” on video experiences also continue to evolve, at anygiven point in time (and on any given hardware/software system) thereare in fact some limits as to what can be programmed into a digitalenvironment experience such as a video game. These limits can sometimesrequire a balancing of competing factors, so as not to overtax thehardware/software in a way that completely locks up the display/programor makes it so “slow” or “laggy” that the user's experience isnegatively affected. In video games, such factors can include speedand/or complexity of the action within the environment and/or by thegame character (i.e., the ability to do more complicated moves, etc.),as well as the “cinematography” of the user's experience (i.e., toprovide a high degree of visual “immersion” of the user into the gameexperience, such as by affording the user “control” of the camera). Withthe evolution of narratives (or “story lines”) in video games, forexample, cinematography has become even more important in video games asa means to convey story and emotion. To improve the visual fidelity ofvideo games, camera systems are being developed for those games that canprovide increased cinematic capabilities (that can make the graphicalexperience more like a movie). However, the speed and/or complexity ofthe player's action (or other virtual things with which the playerinteracts in the environment) can push or reach the aforementionedlimits on hardware/software/etc., and therefore can require that camerashots become more conservative, again preferring function over form (forexample, fewer close-ups, less richness of detail in the player'ssurroundings, etc.).

As a consequence of dealing with the aforementioned “limits”, prior artvideo games (or similar digitized, interactive virtual or simulatedexperiences) tend to one of two types: (a) those that mostly focus oncinematography but often suffer with “lesser” gameplay, and (b) thosegames that mostly focus on gameplay but often suffer with “lesser”cinematography.

Other factors impact the approach to programming such virtualenvironments and experiences. For example, human users have some limiton their abilities to “multi-task”, and those limits vary across thehuman population to which the video game or other virtual program may bedirected. In 3D video experiences, for example, if a player has todevote too much attention and effort to “controlling” the camera, it candetract from or otherwise negatively impact their ability to focus onthe actual “gameplay” (fighting the bad guys, avoiding various perils inthe game, etc.).

U.S. Pat. No. 6,040,841 to Cohen et al. illustrates one alternativeapproach to camera control within a three-dimensional virtualenvironment. According to its Abstract, the '841 patent teaches to“automatically apply[ ] rules of cinematography typically used formotion pictures. The cinematographic rules are codified as ahierarchical finite state machine, which is executed in real-time by acomputer in response to input stimulation from a user or other source.The finite state machine controls camera placements automatically for avirtual environment. The finite state machine also exerts subtleinfluences on the positions and actions of virtual actors, in the sameway that a director might stage real actors to compose a better shot . .. .” Although this approach provides some benefits of achieving“cinematographic effects”, it is directed to graphically simulating“communication” or talking between virtual actors in the virtualenvironment. As such, it may have some efficacy in applications such aschat rooms or the like (in which the focus is not “fast action”, forexample, but merely having conversations between virtual actors), but asmentioned above it has several shortcomings that make it less thanoptimum for simulated gameplay in an action video game or other suchapplications.

For example, the '841 patent teaches using a finite state machine tocontrol the camera (“in this certain circumstance, here's how the camerashould behave.”). In other words, the camera is limited to one of thestates that has been set up or pre-programmed. Using a virtual chat roomas an example, a user instructs his or her avatar (virtual actor) to goover and have a virtual conversation with another avatar. This isrelatively easy to do in a chat room or party room program orapplication, where you simply have four or five people walking around.It is not easy to do in video games, especially in fast/high action,highly-detailed video games.

As another example of shortcomings of the '841 patent, it teaches to“move” or sometimes even take out (erase from the camera view) one ormore of the other avatars (the ones not involved in the user's chatsession), if that other avatar is in the way of framing the camera viewin an optimal way according to the “rules of cinematography.” Although avideo game could use such an approach, it would be at the cost ofcausing a disruption in the user's “immersion” into the virtual world.

Thus, the '841 patent appears to describe a discrete state of avataraction, and is “action driven”—that is, the user determines which ofseveral discrete states the camera will be in by selecting from a givenmenu of avatar actions. The camera stays in its given state until theplayer executes another action. For example, if the user selects theaction “A talks to B”, the '841 patent camera stays in that specificstate until the user gives another action command (such as “I walkaway”). Within that given single relatively static state, the '841patent system says. “I need to frame ‘A’ talking, almost to theexclusion of everything else.” For example, as noted above, ifthings/avatars are not immediately involved in the action that the userhas selected, the '841 patent system teaches to even remove things fromthe scene, if that helps frame the camera view in an optimal wayaccording to the “rules of cinematography.”

SUMMARY OF THE INVENTION

For the purpose of summarizing the invention, certain objects andadvantages have been described herein. It is to be understood that notnecessarily all such objects or advantages may be achieved in accordancewith any particular embodiment of the invention. Thus, for example,those skilled in the art will recognize that the invention may beembodied or carried out in a manner that achieves or optimizes oneadvantage or group of advantages as taught herein without necessarilyachieving other objects or advantages that may be taught or suggestedherein.

The interactive, cinematic camera system of the invention can helpbalance some of the various design considerations and limitationsdiscussed above, to provide an improved user experience in 3D virtualenvironments such as video games. The invention can help maintain adynamic, artistic, and contextually relevant composition while remainingconducive to gameplay or other interaction with the digital environment.In a preferred embodiment, the camera system is adaptive to the player,while maintaining a vision established by the cinematographer and/orgame designer.

Another description of the balance that can be improved via theinvention: if the cinematography becomes too pre-scripted, theplayer/user does not feel in control; if the camera instead is toopassive, the experience can become dull for the player, and/or can ceaseto be as “cinematic” as it might otherwise be. As described herein, thepresent invention provides an improved balance of those considerations,which is particularly useful in certain applications such as actionvideo games.

In certain embodiments, the present invention provides a new camerasystem which is capable of automatically adapting in desirable ways tothe gameplay/digitized environment. Preferably, this automaticadjustment can occur at all, or substantially all, times during theuser's/player's experience, and thereby can avoid or reduce thecinematic or other limitations or distractions of prior art systems(such as ones requiring user control of the camera or having fixedcamera positions).

In certain embodiments, the present invention provides an “intelligent”or algorithm-driven camera system for third person games, using theplayer's own gameplay movements and actions as input to determine andframe the camera view or scene, without any need for separate user inputregarding the “camera” (e.g., without the player having to independentlyoperate the camera). The algorithm(s) involved can take into account awide variety of factors, including certain cinematographic or other“rules” that can be created and/or selected by a programmer, by the user(such as providing the opportunity for various styles, etc.), orotherwise. Among other things, such an algorithmic approach to cameracontrol can take into account and analyze relevant information in thescene, and then automatically direct/move the camera view experienced bythe user according to the rules within the algorithm(s) or similarprogramming structure. Examples of such “scene information” include theposition of the player-controlled main character, the position of othercharacters in the scene, environmental features, various specialeffects, and the occurrence of special events during gameplay.

In certain embodiments, the camera system is at least “semi-autonomous”,so that certain input from the user can be weighted by the algorithm(s)so as to give the user the sensation of “taking control” of the camera(albeit preferably in a limited fashion and/or for a limited time,because reverting “all” camera control back to the player would reduceor eliminate the desirable “automation” of the camera control that canbe achieved with the invention). As also described herein, by heavilyweighting (valuing) the player's avatar as a point of interest (POI)within the virtual world, a programmer/designer can increase theprobability that the avatar will be in the view that is selected anddisplayed to the player. This can be very useful in many applications ofthe invention, such as the action video game systems used within certainexamples described herein.

The “programmability” of the camera control can be varied, and cancombine multiple concepts that a game designer may deem desirable.Examples include obstruction-correcting cameras that adapt according tothe nature of the environment in order to allow for the best shotpossible. The system also can include emotionally aware and expressivecameras that react according to the emotions of the character, and themood of the scene. For example, if a character's emotional involvementis low, the camera shots can be programmed to be long (such as using awide field of view and being relatively further from the subject); ifhis emotional involvement is neutral, the camera shots will be mediumsize/speed; and if the character has high or subjective emotionalinvolvement, the camera shots will be low angle and medium shots. By wayof further examples, the system of the invention can includedialogue-driven cameras that understand the rules of cinematography in adialogue setting (e.g. complimentary angles, 180 degree rule, subjectivevs. objective, etc.), for screen situations in which multiple characterstalk to each other or are otherwise “together”. Preferably, however, thepresent invention uses an approach such as a “state stack” or “modifierstack,” so that “rules” (such as the rules of cinematography) do nothave such “absolute” control over camera view framing and behavior.

The present invention also allows programmers and designers to “tag”and/or apply a “weight” or value to a virtually unlimited set of “pointsof interest (POIs)”, and make those POIs available for possibleinteraction with the user's avatar or other purposes. In certainembodiments, it can provide a substantially dynamic virtual interaction,such as by reevaluating the camera shot on a virtually constant basis.

These and other objects, advantages, and embodiments of the inventionwill become readily apparent to those skilled in the art from thefollowing detailed description of the preferred embodiments havingreference to the attached figures, the invention not being limited toany particular preferred embodiment(s) disclosed.

BRIEF DESCRIPTION OF THE DRAWINGS

FIG. 1 is a block diagram or flowchart illustrating certain aspects ofan embodiment of the invention, entitled “Camera Position/MovementLogic”;

FIG. 2 is a block diagram or flowchart illustrating certain aspects ofan embodiment of the invention, entitled “3-Point Iterative Algorithm”;

FIG. 3 is a block diagram or flowchart illustrating certain aspects ofan embodiment of the invention, entitled “Choosing a New Camera”;

FIG. 4 is a block diagram or flowchart illustrating certain aspects ofan embodiment of the invention, entitled “Scoring a Camera”;

FIG. 5 is a block diagram or flowchart illustrating certain aspects ofan embodiment of the invention, entitled “Finding Best Point of View(POV)”; and

FIGS. 6A, 6B, and 6C are different perspectives of an illustrativeembodiment of the invention, illustrating a user's avatar 10, other POIs(12, 14, 16, and 18), and a plurality of cameras (22, 24, 26, 28, 30,and 32). FIG. 6A is a perspective view taken from nearly straightoverhead; FIG. 6B is an elevation perspective view taken along line6B-6B of FIG. 6A; and FIG. 6C is similar to FIG. 6B but taken from aslightly higher position and angled downwardly.

DETAILED DESCRIPTION

Embodiments of the present invention will now be described withreferences to the accompanying Figures, wherein like reference numeralsrefer to like elements throughout. The terminology used in thedescription presented herein is not intended to be interpreted in anylimited or restrictive manner, simply because it is being utilized inconjunction with a detailed description of certain embodiments of theinvention. Furthermore, various embodiments of the invention (whether ornot specifically described herein) may include novel features, no singleone of which is solely responsible for its desirable attributes or whichis essential to practicing the invention herein described.

Although the methods of the invention are described herein with stepsoccurring in a certain order, the specific order of the steps, or anycontinuation or interruption between steps, is not necessarily intendedto be required for any given method of practicing the invention.

As indicated above, although much of the description herein focuses onapplications such as action video games, persons of ordinary skill inthe art will understand that the invention has utility in a broad rangeof applications other than games. Three-dimensional architecturalrenderings, virtual tours of art galleries and/or other institutions,real estate websites or similar displays, and other interactive virtualenvironments are just some of the many examples of such applications.Thus, persons of ordinary skill in the art will understand that variousterms used herein may have similar or identical concepts in industriesand applications other than video games.

Likewise, persons of ordinary skill in the art will understand the basicconcepts of constructing virtual 3D environments and avatars, havingthose avatars interact with those environments, and using cameras withinthat environment to provide the human player(s) an interface into thosevirtual worlds. They also will understand that, although much of thedescription herein refers to avatars, certain applications may not use“avatars” (or at least not ones that are visible). For example, they mayattempt to give the user the illusion of “first person” travelingthrough the virtual world. In these cases, the system preferably willframe the shot based on the “non-avatar” POIs that are in the relevantscene or area of the virtual environment.

Some of the basic logic and concepts involved in the methods andapparatus of the present invention can be appreciated by reference tothe attached drawings. As shown in FIGS. 6A, 6B, and 6C, an avatar 10 istypically designated as one of several points of interest (POIs) withinthe virtual environment. As further explained below, by “weighting” theavatar sufficiently (i.e., making the avatar of sufficient“importance”), the programmer or designer can make it more likely (orpossibly even “certain”) that the camera that is selected (to bedisplayed to the human user) will be one that includes a view of theavatar. Other POIs are shown generically at locations 12, 14, 16, and18. Although these are illustrated as other “person” avatars, asindicated elsewhere these POIs can be any desired “thing” in the virtualenvironment, including fixed or movable elements of scenery, pathways,enemy weapons or avatars, etc. Each of these elements preferably can becreated and assigned a “weight” by the programmer or designer, toprovide the desired gameplay or other interaction for the human user.

A ring of cameras 20 is also illustrated in FIGS. 6A, 6B, and 6C, asincluding cameras 22, 24, 26, 28, 30, and 32. Persons of ordinary skillin the art will understand that any suitable number and arrangement ofcameras can be used within the invention depending on the specificapplication, and that the cameras can be (among other things) programmedso that they “travel” with the player's avatar as it moves through thevirtual world. Among other things, cameras might not completely surroundthe avatar, might not be coplanar with each other, may be distributedaround the avatar at equal intervals/angles from each other, etc. By wayof example of non-coplanarity, camera 28 is positioned at about the“floor” of the environment, so that it generates an upwardly angledshot. In contrast, camera 22 is positioned to shoot from about “chestheight” of the avatar. As discussed elsewhere, in many applications“above and from behind the avatar” will be a preferred/useful cameraposition.

Persons of ordinary skill in the art also will understand that, in manyvideo games and other applications, the user's avatar “moves” throughthe virtual world, either at the user's direction or otherwise.Accordingly, the positional relationship of the various elementsillustrated in FIGS. 6A, 6B, and 6C can be very dynamic, if (forexample) a player's avatar is running, jumping, spinning, or otherwiseengaged in various “movement” activities within the virtual world.

During the interaction of the player/avatar within the virtual world,the view displayed to the human user can be from any of the camerasillustrated in FIGS. 6A, 6B, and 6C. Various embodiments of theinvention include methods and apparatus for providing a dynamic andautomated selection from among those cameras, to provide to the user adesired visual experience to enhance the user's interaction with thevirtual world. Depending on the particular application, that experiencecan be customized across a wide range of balancing of functionality andaesthetic experience for the user.

In certain embodiments of the invention, the overall logic of the cameraselection, position, and movement can be illustrated as shown in FIG. 1.This method preferably includes one or more of the steps, methods,and/or apparatus illustrated in that Figure.

As indicated in block 100, the game/system display frame isupdated/rendered on a periodic basis (typically many times/second).Those rapid updates/renderings can each be slightly different from eachother, resulting in the illusion (to the human eye) of movement.Preferably, this game display refresh rate can occur at a differentand/or varying time interval from the frequency of the updatecalculations of the movement damping procedure (such as executing theAlgorithm of FIG. 2). Also preferably, the movement dampingprocedure/calculation (see FIG. 2) occurs on a fixed/constant timeinterval, to permit desirable control of the speed and/or accelerationof the camera as it moves through the virtual environment, so as toavoid “jarring” the human user. Such “jarring” or other negativeexperiences can occur if a human user is confronted with displays ofjerky or “too-rapid” camera movements.

Persons of ordinary skill in the art will understand that, although theexample described in reference to FIG. 1 is “initiated” by each displayframe update, other embodiments can be triggered by other events.Preferably, the process is triggered as often as is required to maintainvisual quality within the simulation, as perceived by the human users ofthe system. Examples include, without limitation, initiating the variousprocesses of the invention based on a fixed timer/interval, a randomtime generator interval (perhaps constrained within certain timelimits), etc.

As indicated in block 102, each time the system frame renders/updates,check: does the current camera provide a clear line of sight to thetarget? If it does, the Time Counter is cleared (set to zero) as atblock 110. If not, increment the Time Counter (accrue the time/gameupdate frames that the camera has been at the current location/point)(see block 104), and then determine whether the Time Counter has reachedits preset limit (block 106). If it has not, the logic returns or loopsback to await the next display frame update (block 100) which kicks offthe cycle again.

If instead the Time Counter has reached its preset limit, this typicallyindicates that the camera has not moved for a number of frames. Althoughsuch a failure to move can result from a player simply sitting idly(rather than moving his avatar), it also can result from other causesthat could “lock” the system logic. Using the video game example, if acamera is trailing behind an avatar and the avatar goes through adoorway, if the door shuts before the camera makes it through thatdoorway, certain “reality rules” might stop the camera from being ableto follow the avatar through that door (e.g., cameras normally can'ttravel through solid objects such as doors). To handle such situations,once the Time Counter limit has been reached (with no camera movement),the system logic preferably disables or turns off the relevant rules(such as the aforementioned prohibition against collisions or movingthrough solid objects). This is illustrated in block 108 of FIG. 1.Preferably, this disabling is only for the current update cycle, and therule is “reactivated” for the next frame display update cycle. In otherwords, in such embodiments and in such situations, the camera ispermitted to “break” certain “rules” and move in ways that normallywould not be permitted by the system.

As indicated in block 112 of FIG. 1, the logic preferably next selects acamera. This can be done with any suitable method, including theexemplary method illustrated in the “Finding the Best Point of View(POV)” calculation process illustrated in FIG. 5.

In order to avoid “jarring movement” of the camera (which coulddisorient or, in the extreme, even nauseate the user), the systempreferably includes one or more means for “dampening” the movement thatmight otherwise occur. In the embodiment illustrated in FIG. 1, aninterpolating damping procedure is used (such as the one illustrated inthe Algorithm of FIG. 2). This interpolator preferably uses a “3-point”iterative algorithm, using the following three points for eachiteration: (1) the “current point/location” at which the camera islocated at the time of the calculation; (2) an “ideal point/location”which is the point selected as the “Best POV” (calculated in block 112);and (3) a “desired point/location” which is a point between the“current” and the “ideal” points, and which is used as a dampening linkbetween the two. For certain situations (such as when the player'savatar is still and there are no “targets” around or enemies approachingwithin the virtual world), the three points can be at the same location.Once something forces the ideal point to move away from the currentpoint, however, the algorithm can be used to help calculate the camera'sposition and movement, for an improved player experience.

In order to generate a more “normal” sensation of movement for the user,the camera preferably does not immediately track or move to the “idealpoint”. Instead, the 3-point iteration allows the “current” point of thecamera (the one being viewed by the user) to gradually accelerate towardthe ideal point, and gradually slow down and stop at the ideal point (ifthe user/avatar ever catches up with the “ideal point”). Preferably, theinterpolator calculates a proposed (1) movement of the “desired point”of the camera toward the “ideal point” by a distance determined by apreset “desiredSpeed” variable; and (2) movement of the “current point”of the camera toward the “desired point” by (a) the distance between thecurrent and desired points multiplied by (b) a preset “sponge factor,”which is another variable that can be set by the programmer or designer(see block 114).

In addition, a number of “movement modifiers” can be programmed into a“modifier stack” of factors that the logic considers during each cycleof FIG. 1. Examples of “post interpolation” modifiers are shown asvarious blocks within area 120. As shown in block 122, one such modifiercan focus on whether the proposed camera movement (calculated by theinterpolator in block 114) will block the camera's view of the target.If it will, that or another modifier can move the proposed cameraposition sufficiently close to the target so that camera's view is NOTblocked (see block 124). If the view is not blocked, the system canproceed to another modifier such as block 126, which can check forcamera collisions with geometry or blocking volumes (stop at collidingobject).

Other examples of modifiers are shown in blocks 128, 130, and 132. Inblock 128, a modifier logic monitors whether the proposed movement ofthe camera will place it too far from the target. If yes, an “auto snap”or similar logic can be used (see block 130) to force the camera to“snap” to a sufficiently close distance to meet whatever parameters havebeen programmed in that regard. In block 132, a modifier can determinewhether to push the camera up to a minimum distance above the “floor” orother surface of the virtual environment.

Persons of ordinary skill in the art will understand that the “modifierstack” concept of the invention can include modifiers that are appliedafter the interpolator calculation, before that calculation, or both. Ifafter (or “post”) modifying the calculated position, certainapplications of the process can be described as “massaging” thecalculated camera position.

Finally, after selection of the camera, interpolation calculations, andapplication of any modifiers, the “new” position of the camera isdetermined (see block 134), and the program preferably moves the camerato that position. Depending on the application and the circumstances,the “move” actuated in block 134 could be a move of zero distance.

Persons of ordinary skill in the art will understand that the foregoingmodifiers are merely examples, and that virtually any desired factor canbe incorporated into the “modifier stack” to affect the camera movement.As previously mentioned, these modifiers can even include some degree of“camera control” by the user (although preferably the user is nevergiven complete camera control, as that would remove many of the benefitsthat can be provided by the invention). Moreover, the order ofapplication of the modifier calculations can be varied to suit theparticular application for which the invention is being used, and theremay not be any modifiers at all in certain applications.

As shown in block 112 of FIG. 1, the system preferably “automatically”selects a camera from among various potential cameras. An example ofsuch a selection process is illustrated in FIG. 5, entitled “Finding theBest Point of View (POV).” For each predefined camera that is a possiblePOV in the current situation in the virtual environment, the systempreferably calculates a basic camera score for the POV (see block 180).That calculation can be accomplished in any suitable manner, includingby way of example via the process illustrated in FIG. 4.

Once that basic score has been calculated, the score can be further“adjusted” for other factors. In block 182, for example, the score of aPOV can be “penalized” or discounted based on the amount of rotationthat would be required relative to the current camera. Because largeswings in camera orientation can be disorienting to a user, typicallythe programmer will discount the score further as the necessary“orientation swing” increases (although the example of FIG. 5illustrates only a three-tiered discounting scheme, persons of ordinaryskill in the art will understand that any number of stepped discounts orother approaches could be used within this step of the process). Onceany such “discounting” (or alternatively, “enhancement”, if a programmeruses some factor(s) that he/she deems make the camera more desirable)has been applied to the camera score, that score is compared to theprevious “best” score (of those calculated during this cycle of “findingthe best POV”) (see block 184), and if better than that score, itsassociated camera position is saved/stored (see block 186). If the scoreis not better than that previous score, the score and its associatedcamera position are discarded (again, for purposes of this specificcycle of “finding the best POV”; see block 188).

One of the many approaches to the camera scoring discussed above isillustrated in FIG. 4. For example, for applications such as video games(or other “target-rich” applications), a camera's score can be increasedfor each valid target that would be displayed if that camera were to beselected (see block 190). Other “modifiers” can be programmed in thescoring, such as the one illustrated in block 192 (will the camera bepushed closer to the player by terrain or by camera blocking volumeswithin the virtual environment?). If yes, the camera's score can bedecreased, such as indicated in block 194 (by the percentage by whichthe camera will be pushed in).

Instead of a finite state machine approach such as taught in theaforementioned '841 patent, the present invention preferably uses anapproach such as a “state stack” or “modifier stack.” Although thecamera has a “base” behavior that is determined by the state, that stateis determined only in a very simple manner. For example, the camera canbe constrained to be a chase camera or a rail camera (FIG. 3 and FIG. 5illustrate examples of such base behaviours). Beyond that simple basestate or base behavior, the present invention can use a modifier stackso that, for example, any action that a user imparts to his avatarcauses the program to move through a series of modifiers that travelwith the camera. This provides a much more dynamic feel for theresulting video display seen by the user.

Preferably, the present invention also allows a programmer or designerto tag points of interest (POIs) within the virtual world, and usesthose POIs dynamically to calculate and select a camera for display tothe user, the position of the camera, and other things. The presentinvention also preferably reevaluates the camera shot on a virtuallyconstant basis—such as every 1/30 of a second. This gives the user theimpression that the shot is constantly moving as the user moves throughthe virtual world. In effect, this system virtually constantly evaluatesthe camera position, the player position, and the positions of the POIs,all relative to each other on a per frame basis (approximately 30 Hz).

For many applications of the present invention, programmers anddesigners will constrain the camera apparatus and methods so that itcannot “remove” anything from the virtual scene (just as things do notspontaneously move or disappear in the real world). Among other things,such changes would or could disrupt the user's sense of continuity andimmersion into the virtual world (for example, if the camera were tosuddenly cut from one location to another without any action on theuser's part).

In contrast to the “cinematographic events” taught in the aforementioned'841 patent, preferred embodiments of the present invention do not havea module that (a) determines what kind of an “event” is occurring, andthen (b) passes that information to another module. Instead, the presentinvention preferably depends on the mode picked by the game designer.The designer can establish a number of POIs (e.g., things about whichthe game designer has determined that the game should know). Typically,these POIs can be things of relevance to the eventual player of thegame. In addition to “mobile” items such as avatars that can movethrough the virtual world, these POIs can include other mobile items(such as enemy targets) as well as items that have relatively “fixed”positions within the virtual world.

As a player navigates through the virtual world, the camera preferablytakes into account points of interest, and attempts to frame the camerashot appropriately based on the weight that the game designer has givento the various points of interest (POIs), using the programming logic ofthe modifier stack or similar tool.

As will be understood by persons of ordinary skill in the art,programmers and/or game designers can use any suitable hardware and/orsoftware tools to practice the invention. These include not onlypersonal computers and handheld or other gaming systems, but morebroadly tools such as any suitable programming languages, platforms,coding programs, rendering engines, and many others. At the presenttime, examples of such tools and languages include C, C++, and Java,consoles from Nintendo, Sony, Microsoft, or others, PCs or Apple (orother computers can be used), etc. The specific algorithms, hardware,console, and other “forms” of the invention are virtually unlimited.

In other words, the rendering engine, platform/console, and languageused to practice the invention are arguably immaterial. Instead, some ofthe main features of the invention that can be practiced in manydifferent ways include having a three-dimensional rendering engine,points of interest (POIs) of what you want to display within the virtualenvironment, and a camera view into that virtual world. The logic,apparatus, and techniques of the invention can be adapted to anysuitable programming language, platform, or other aspect of presentingand/or interacting with three-dimensional virtual environments.

In many applications and embodiments, the present invention can beimplemented by the game designer or programmer selecting a single state(either programmatically, or through use of a game design tool) from oneof a preferably small number of states, such as three states. Althoughcertain embodiments of the invention could include a larger or even“large” number of states, a small number of states is easier to programand much more manageable than having to code many specific behaviours.Typically, the state chosen can provide a base behavior or motion forthe camera. For example, in a wide open area of the virtual environment,a chase camera may be preferred, while in an enclosed space within thevirtual environment, a camera that is constrained to a rail might bebetter suited (might be more likely to provide a desired gameplayexperience for the user). Preferably, whichever of the states isselected, that state can handle and implement virtually any action bythe user within the scene. The camera preferably also can take intoaccount all of the relevant points of interest (POIs) as part ofautomatically determining the camera view, by using the modifier stack(the programming that “travels” with the camera) or similar technology.

Regarding the use of “cinematographic rules” or similar concepts (toachieve heightened cinematographic effects, for example), the presentinvention preferably includes some or all of those rules, but uses themonly as guidelines. For example, for applications of the presentinvention involving an action video game, the invention preferably willnot remove certain objects from the camera shot or automatically “move”or otherwise cause a discontinuity in the virtual world by adjusting theposition of one of the people or objects within the camera shot. Asanother example, in many applications of the invention, theprogrammer/designer will attempt to avoid “cutting” any of the actionwithin the virtual world. This is true even if such cutting would bemore true to the cinematographic rules. Thus, the present inventionsometimes overrides the cinematographic rules with certain otherprinciples (such as the idea that you don't want to disorient a playerby having certain objects suddenly disappear or be moved to a differentposition, without having had any relevant input from the user).

Thus, at least certain embodiments of the present invention can holdcertain principles as being more important than the aforementioned“cinematographic rules.” These additional principles can include, by wayof example and not by way of limitation, not disorienting the humanplayer, not allowing things to be removed from the camera shot, makingit a priority to keep the player's avatar on screen (in the selectedcamera shot), etc. In other words, the technology commonly used inmovies (following the cinematographic rules) is different from thetechnology typically required in action video games (such as ones thatcan be created with the present invention). Said another way, actionvideo games are a different medium than movies or the video technologiesin which the '841 patent would be useful.

In certain embodiments, the present invention can use cutting ortweening to define motion from one camera position to another. Cutsprovide an instantaneous transition from one view to another, but tendto disrupt gameplay. Tweening can be accomplished with, for example, a3-point iterative calculation. In a preferred embodiments, the threepoints can be: the ideal position, desired position, and currentposition. In such embodiments, the ideal position as determined by therest of the system can move to any location at any time. The desiredposition steps in a linear fashion in the direction of the idealposition, and the current position steps some fraction of the distancebetween it and the desired position. At rest (when the player/characteris not moving within the virtual scene), all points are in the samelocation. During motion, however, the camera of the invention preferablyautomatically accelerates from rest, decelerates to rest, and smoothlydeals with a dynamically changing target.

One embodiment of a preferred motion of the ideal position can bedescribed using a number of tools. In certain embodiments of theinvention, the various degrees of freedom of the camera motion can beindependently constrained. In certain embodiments, the motion can beconstrained to a point, spline, or plane, and the camera target(viewpoint) and actual camera can both move independently using the samealgorithm. In some embodiments, functions describe the possible pathsthat can be taken from rotation around targets, to linked positions ongeometric shapes where the camera position is derived from the positionof the target.

In certain embodiments of the present invention, when the camera isunconstrained, it can use the aforementioned points of interest (POI) todetermine the ideal location and rotation. Using a weighting schema (forexample, a schema that takes into account attributes like the locationon or off screen, angle off axis, unobstructed visibility, and/or otherfactors), both the current frame and a number of possible frames areevaluated and the highest score is determined to be the best position.Rulesets then determine the method of transitioning between the currentand new best position, choosing a method of motion that does not breakthe rules of cinematography (cutting across the axis, tweening overhead,etc). The resultant camera motion of the invention provides a unique“cinema style” look and feel to an interactive experience such as anaction videogame.

In accordance with an exemplary embodiment of the present invention, thepresent video game camera system apparatus automatically changes theapparent moving direction of the camera and/or modifies the apparentcamera angle depending upon the controlled character's circumstance(e.g., he is inside a room, outside a room, on a ledge, behind a wall,running, jumping, swimming, scared, excited, isolated, anxious,surprised, etc.), the position of other characters in the scene,environmental features, various special effects, and the occurrence ofspecial events during gameplay. If the camera system detects that, forexample, a wall exists between, for example, the player controlledcharacter and a camera point of view, a calculation is made as to therequired camera movement to prevent the obstruction between the eye ofthe camera and the object. The camera is then automatically moved to anew position in accordance with the calculated moving angle. The cameraperspective is automatically changed to select the best camera angleaccording to the character's circumstances so that the player can enjoythe visual effects being experienced in the three-dimensional worldwithout having to control the camera him/herself.

In another exemplary embodiment of the invention, a video game systemincludes a control processor for playing a video game including a gamecharacter controlled by a player. A camera system apparatus communicateswith a camera and determines the direction of movement of the cameraand/or modifies the apparent camera angle depending on the playercontrolled character's circumstance. The position of the camera ismodified during gameplay according to occurrences in the game, wherein amodifying amount is determined based on various factors, such as thecharacter's circumstance, the position of other characters in the scene,environmental features, various special effects. As indicated above, themethods and apparatus of the invention are useful for a wide variety ofthree-dimensional virtual environments. Certain such video gameenvironments can be described as having “targets” or points of interest(POIs) that the programmer/designer can “tag” or otherwise mark or usefor possible interaction with the user's avatar or for other purposes.

For some “target rich” environments (such as shooting games, forexample), the invention can be practiced by using a specializedweighting system to determine the ideal camera position. Under such anapproach, and as illustrated in FIG. 3 (Choosing a New Camera), one“Post” Modifier within the “modifier stack” can check the area aroundthe player's avatar (within the virtual world) for targets, and if anyare found, can evaluate the best camera angle. This check or sweep isillustrated as logic/method steps and/or apparatus 50, and it can beconfigured or structured on any desired basis, including by way ofexamples, checking out into the virtual environment to a certain radiusfrom the player/avatar, checking for certain types of targets, etc., oreven combinations of such criteria.

If there are no targets within range (and/or that meet any otherspecified criteria), then the camera positioning system falls back uponthe other PostMods in the programming stack (as illustrated bylogic/apparatus 60). However, if the sweep or check locates one or moretargets (or a predetermined minimum or maximum number of targets, forexample), the PostMod can evaluate a number of possible alternativecamera views and, if the analysis of those views shows that any issuperior to the current view (based on various factors and criteria thatcan be established on a customizable basis and used to “score” eachcamera, as illustrated in the example of FIG. 4), the system selectsthat “better” camera view. Typically, the alternative camera views thatget evaluated are generally spread around a circle which is centered onthe player's location within the virtual environment. For applicationsother than action video games, the arrangement of potential cameras canbe any suitable configuration.

In some applications, for example, each camera view can be scored basedon the number of targets the camera would have on screen and multipliedby the “weights” that the programmer/designer has assigned to each ofthe targets. In passing, for many video games, it is useful to programthe player's avatar as a POI and weight it very heavily, so that thesystem will be biased heavily toward including the avatar within theselected camera view.

As mentioned above, the evaluation or scoring of each camera's view alsocan take into account potentially “negative” factors. Such factorsinclude if there is any piece of virtual geometry or a combat camera“blocking volume” blocking the player. The score is further reduced bythe percentage of the distance the camera must push forward in order tobe closer to the player than the collision it detected. In certain ofthe embodiments discussed above, each camera view is then furtherpenalized based on its orientation to the current view. Camera viewsthat are facing forward are worth their full score, views facing toeither side are worth 50% of their total score, and the views facingbackwards or opposite to the current camera view are worth 25%. The bestcamera view out of the set is stored via a vector from the target to thesuggested camera location and is used when the camera updates itsposition in a later PostMod process.

In a preferred game application, the algorithm of FIG. 2 scores thecurrent camera view, and if any enemies are within range it calls forthe ring of camera views to be scored to find out if there is a bettercamera position. Otherwise, the system determines the camera position byevaluating the remainder of the PostMods in the modifier stack.

In certain embodiments of the invention, each character or avatar canhave a number of dynamically-updated cameras whose position and rotationchange depending on the location of the character in the world. This canprovide to a programmer or designer a large numbers of potential camerasfrom which to choose at any given time, and the camera can be selectedby the Best Point of View algorithm (discussed above). Such embodimentsare analogous to a major sporting event where there are many camerasplaced throughout the venue, all simultaneous providing a different viewof the action, with a coordinator (here, the logic of the variousalgorithms and the modifier stack) making a decision about which shotbest frames the current action.

The algorithm illustrated in FIG. 3 scores a single camera view by goingthrough the list of targets, validating those targets, and adding theirscore to the camera's total. It also applies a penalty to the cameraview's score when it detects a collision between the camera location andthe target (which may be the player/avatar). After all of thesecalculations, it returns the score.

As discussed above, in certain embodiments the camera's movement iscontrolled by a dampening system such as a three-point interpolationsystem. The “movement dampening” can help provide smooth camera movementwhile track a moving target (the player/avatar) whose velocity isneither constant nor straight. As indicated above, the interpolationalgorithm uses three points or values:

-   -   Current Point/Value/Location: Where the interpolated value        actually is. This value is always used in calculations to        achieve smooth motion.    -   Middle or “Desired” Value: This point moves a fixed distance at        a fixed frequency. For example, it may step 10 world units ever        1/60th of a second. This point moves in a straight line towards        the final (or ideal) value or point.    -   Final/Ideal Value: This is the final location to where the        interpolator is trying to go. Preferably, this value can        immediately snap to any location, because the system protects        the user against experiencing anything other than a smooth        motion.

In many applications, the designer can manually place “volumes” into thevirtual environment using an offline editor. These volumes encompass agiven area of the virtual world, and can provide a means for thedesigners to give information to the camera system.

-   -   Property Volumes: these can allow for basic properties of the        camera to be modified when the target is within a volume. (such        as target and camera offsets, FOV, Speeds, Sponge Factors, etc)    -   Trajectory Volumes: these can allow for the camera to be forced        to face a given direction (but still aimed at the target).    -   Point of Influence Volumes: give the ability to specify an actor        (the influence actor) to focus the camera on in addition to the        target. Each Point Of Influence Volume has a Point Of Influence        Circle. When the target enters this circle and nears the centre        of the circle, the camera will progressively focus on the        influence actor. An option also allows the camera to always keep        the target in view.

As mentioned above, PostMod's preferably can be applied in a “stack”form, allowing a designer to push and pop various modifiers. In oneembodiment, each PostMod stands alone as a single task to beaccomplished. This allows combinations of various modifiers to influencethe camera behavior, and also allows the camera to have a sense of“state” so that transitioning to these styles or modifications istransparent. Examples of “PostMod” or other Modifications that includethe following, although the system is flexible enough to allow a widevariety of additional modifiers beyond and/or instead of those listedhere:

-   -   Cornering: This post mod will physically rotate the camera        around corners.    -   Basic Properties: This will apply some offsets which can be read        in from configuration file    -   Styles Post Mod: The camera can have “styles” on it which are        read in from a configuration file. Each style is a collection of        properties which will be applied.    -   Point Of Influence: This will watch for volumes that the player        stands in which have a point of interest attached to it. The        camera will rotate towards this point to show it. (placed in        editor in advance)    -   Properties Volumes: Allows the properties of the camera to be        changed on the fly when inside of a particular volume (placed in        editor)    -   Trajectory Volumes: Forces the camera to align to a certain        direction along a combination of the x/y/z axis. (placed in        editor)    -   Camera Aiming: This allows a button press to cause the camera to        snap to be behind the target and allow a “free look” on the        thumbstick to look around    -   In a preferred application, the camera's base motion uses two        interpolators that track the current camera's position as well        as the target's location. Typically, the target location tracks        the root of the character, although as the character animates        through the world, this motion can be erratic. To dampen the        erratic aspects of this motion, designers and programmers can        apply additional interpolators.

In certain video game or similar applications, when the camera is notconstrained to a spline, plane or a point, the ideal position of thecamera typically can be some distance away from the player with a targetat the player's location. In addition, there is a target and cameraoffset which preferably are specified in the camera's local space. Theseare added on to the base locations to give the camera additional heightand rotation.

When transitioning to and from placed cameras, the generic positionlogic (including any PostMod stack logic) preferably is applied. In someembodiments, the camera also can have the ability to cut immediately tothe new location.

As mentioned above, the system preferably includes a means or method ordampening the “virtual movement” of the camera that is experienced bythe human user. Although other dampening approaches can be employed, theexample of the drawings uses a “3-Point Iterative Calculation Algorithm”(or “3PICA”). As illustrated in FIG. 2, such an approach can includeaccumulating the “unused” delta time for each update/rendering of thedisplay/frame (block 200). As indicated in block 202, if thataccumulated time is at least as great as the update frequency that hasbeen programmed for the 3PICA itself (which commonly is set at or around1/60th of a second), the system preferably returns to block 200 toaccumulate further time as part of the next update/rendering of thedisplay/frame.

Once sufficient time has accumulated for the logic to pass through block202, block 204 illustrates that the system determine the number ofcamera position updates that can be achieved within that accumulatedtime. This can be conveniently done, for example, by taking the largestwhole number y that results from dividing the frequency of the 3PICAinto the accumulated frame time (or “delta time”). For y number oftimes, the 3PICA then iterates or cycles through the two calculationsshown in Iteration Loop 214 (blocks 206 and 208). The logic illustratedin block 206 calculate a line from the desired/middle point or valuetowards the ideal/final point or value, and moves the desired/middlepoint or value “desired speed” units in that direction. It also ensuresthat the desired point does not “overshoot” the ideal/final point. Thelogic illustrated in block 208 calculate a line from the “current”point/value towards the “desired”/middle value, and moves the “current”value along this line, by a “sponge factor.” This sponge factorpreferably is a value between 0 and 1, and is selected by the programmerto determine the percentage of the calculated distance that the currentvalue/point (the camera's current position) should be moved. Forexample, a sponge factor value of 0.5 means the current point moveshalfway along the line calculated in block 208.

The apparatus and methods of the present invention have been describedwith some particularity, but the specific designs, constructions, andsteps disclosed are not to be taken as delimiting of the invention.Modifications and further alternatives will make themselves apparent tothose of ordinary skill in the art, all of which will not depart fromthe essence of the invention and all such changes and modifications areintended to be encompassed within the appended claims.

1. A computer system for facilitating human interaction in a virtualenvironment, the system comprising: an algorithm-driven camera system,programmed to operate in a fast paced dynamic environment withoutmodifying the state of the objects in that environment, said systemusing the human's input of movements and actions within the virtualenvironment to determine and frame a camera view for display to thehuman, without any need for separate input from the human to operate thecamera.
 2. The system of claim 1, in which at least one algorithm takesinto account cinematographic or other “rules” that can be created and/orselected by a programmer, and then automatically controls the cameraview experienced by the human according to said rules.
 3. The system ofclaim 1, in which at least one algorithm takes into accountcinematographic or other “rules” that can be created and/or selected bythe human, and then automatically controls the camera view experiencedby the human according to said rules.
 4. The system of claim 1 or claim2 or claim 3, in which the system includes a human-controlled maincharacter, and at least one algorithm takes into account and analyzesrelevant information in the virtual scene such as the position of thehuman-controlled main character, the position of other characters in thescene, environmental features, various special effects, and theoccurrence of special events during the human's use of the system. 5.Apparatus for providing interaction between one or more humans and a 3Dvirtual environment, including: a 3D virtual environment, saidenvironment including a plurality of points of interest that arepreselected and weighted in importance by a programmer; at least onemain character within the environment; a display device for displayingthe environment to said one or more humans; a control device by whichsaid one or more humans can control said character, including movingsaid character within the environment; a plurality of cameras that areprogrammed to travel with the main character as that character moveswithin the environment; and a modifier stack module, said moduleincluding the ability to automatically select from among said camerasthe one that will be displayed on said display device to said one ormore humans.
 6. The apparatus of claim 5, in which said modifier stackmodule uses said plurality of weighted points of interest in theautomatic camera selection process.
 7. The apparatus of claim 5,including a dampening module to smooth transitions in the virtual cameramovement displayed to said one or more humans.
 8. The apparatus of claim5, including means for dampening the virtual camera movement displayedto said one or more humans.
 9. The apparatus of claim 5 or claim 6, inwhich said dampening involves an iterative algorithm using at leastthree points within said virtual environment.
 10. A method for selectinga camera view within a virtual 3D environment, including: providing avirtual 3D environment having programmed therein points of interest(POIs) identified as having relative degrees of importance; providing aplurality of camera views into that virtual world; using said relativedegrees of importance of said POIs to select from those camera views thecamera view to be displayed to a human interacting with the virtualenvironment, without requiring human control of said camera view.
 11. Acomputer-readable storage medium having stored therein instructionscapable of causing a computer to perform the method of claim
 10. 12. Amethod for controlling the simulated movement of a camera through avirtual 3D environment, including: providing means for determining anideal viewpoint to display from the virtual environment; providing meansfor determining the camera viewpoint current being displayed; providingmeans for smoothly transitioning the viewpoint being displayed from thecurrent viewpoint toward the ideal viewpoint.
 13. The method of claim12, including using a dampening algorithm to help accomplish the step ofsmoothly transitioning the camera viewpoint.
 14. A computer-readablestorage medium having stored therein instructions capable of causing acomputer to perform the method of claim
 12. 15. A computer-readablestorage medium having stored therein instructions capable of causing acomputer to perform the method of claim 13.